[ATfL] Make sure internal shell is not used by lit #602

pawosm-arm · 2025-10-23T20:50:43Z

This is to mitigate the effects of this commit:
llvm/llvm-project@ee0f86d

This is to mitigate the effects of this commit: llvm/llvm-project@ee0f86d

pawosm-arm · 2025-10-23T20:55:21Z

The error fixed by this commit:

23:40:55  -- Testing: 96738 tests, 4 workers --
23:47:36  Testing:  0.. 10
23:47:36  FAIL: Clang :: Driver/env.c (13142 of 96738)
23:47:36  ******************** TEST 'Clang :: Driver/env.c' FAILED ********************
23:47:36  Exit Code: 1
23:47:36  
23:47:36  Command Output (stdout):
23:47:36  --
23:47:36  # RUN: at line 5
23:47:36  bash -c env | grep LD_LIBRARY_PATH | sed -ne 's/^.*=//p' | tr -d '\n' > /workspace/build/stage/product_build/tools/clang/test/Driver/Output/env.c.tmp.ld_library_path
23:47:36  # executed command: bash -c env
23:47:36  # note: command had no output on stdout or stderr
23:47:36  # executed command: grep LD_LIBRARY_PATH
23:47:36  # note: command had no output on stdout or stderr
23:47:36  # executed command: sed -ne 's/^.*=//p'
23:47:36  # note: command had no output on stdout or stderr
23:47:36  # executed command: tr -d '\n'
23:47:36  # note: command had no output on stdout or stderr
23:47:36  # RUN: at line 7
23:47:36  env -i LC_ALL=C LD_LIBRARY_PATH="/workspace/build/stage/product_build/lib:/workspace/build/atfl/lib:/workspace/build/atfl/lib/aarch64-unknown-linux-gnu:." CLANG_NO_DEFAULT_CONFIG=1    /workspace/build/stage/product_build/bin/clang /workspace/src/clang/test/Driver/env.c -### -o /workspace/build/stage/product_build/tools/clang/test/Driver/Output/env.c.tmp.o --target=i386-unknown-linux      --sysroot=/workspace/src/clang/test/Driver/Inputs/basic_linux_tree      --rtlib=platform --unwindlib=platform -no-pie      2>&1 | /workspace/build/stage/product_build/bin/FileCheck --check-prefix=CHECK-LD-32 /workspace/src/clang/test/Driver/env.c
23:47:36  # executed command: env -i LC_ALL=C 'LD_LIBRARY_PATH=%{readfile:/workspace/build/stage/product_build/tools/clang/test/Driver/Output/env.c.tmp.ld_library_path}' CLANG_NO_DEFAULT_CONFIG=1 /workspace/build/stage/product_build/bin/clang /workspace/src/clang/test/Driver/env.c '-###' -o /workspace/build/stage/product_build/tools/clang/test/Driver/Output/env.c.tmp.o --target=i386-unknown-linux --sysroot=/workspace/src/clang/test/Driver/Inputs/basic_linux_tree --rtlib=platform --unwindlib=platform -no-pie
23:47:36  # note: command had no output on stdout or stderr
23:47:36  # error: command failed with exit status: 127
23:47:36  # executed command: /workspace/build/stage/product_build/bin/FileCheck --check-prefix=CHECK-LD-32 /workspace/src/clang/test/Driver/env.c
23:47:36  # .---command stderr------------
23:47:36  # | /workspace/src/clang/test/Driver/env.c:20:17: error: CHECK-LD-32: expected string not found in input
23:47:36  # | // CHECK-LD-32: "{{.*}}ld{{(.exe)?}}" "--sysroot=[[SYSROOT:[^"]+]]"
23:47:36  # |                 ^
23:47:36  # | <stdin>:1:1: note: scanning from here
23:47:36  # | /workspace/build/stage/product_build/bin/clang: error while loading shared libraries: libc++.so.1: cannot open shared object file: No such file or directory
23:47:36  # | ^
23:47:36  # | <stdin>:1:140: note: possible intended match here
23:47:36  # | /workspace/build/stage/product_build/bin/clang: error while loading shared libraries: libc++.so.1: cannot open shared object file: No such file or directory
23:47:36  # |                                                                                                                                            ^
23:47:36  # | 
23:47:36  # | Input file: <stdin>
23:47:36  # | Check file: /workspace/src/clang/test/Driver/env.c
23:47:36  # | 
23:47:36  # | -dump-input=help explains the following input dump.
23:47:36  # | 
23:47:36  # | Input was:
23:47:36  # | <<<<<<
23:47:36  # |             1: /workspace/build/stage/product_build/bin/clang: error while loading shared libraries: libc++.so.1: cannot open shared object file: No such file or directory 
23:47:36  # | check:20'0     X~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ error: no match found
23:47:36  # | check:20'1                                                                                                                                                ?                  possible intended match
23:47:36  # | >>>>>>
23:47:36  # `-----------------------------
23:47:36  # error: command failed with exit status: 1
23:47:36  
23:47:36  --
23:47:36  
23:47:36  ********************
23:57:24  Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90.. 
23:57:24  ********************
23:57:24  Failed Tests (1):
23:57:24    Clang :: Driver/env.c
23:57:24  
23:57:24  
23:57:24  Testing Time: 984.80s
23:57:24  
23:57:24  Total Discovered Tests: 130955
23:57:24    Skipped          :   219 (0.17%)
23:57:24    Unsupported      : 45532 (34.77%)
23:57:24    Passed           : 85110 (64.99%)
23:57:24    Expectedly Failed:    93 (0.07%)
23:57:24    Failed           :     1 (0.00%)
23:57:24  FAILED: CMakeFiles/check-all /workspace/build/stage/product_build/CMakeFiles/check-all

This reverts commit 32de3b9. This relands the enablement of the internal shell for clang by default. This was reverted last time for some Z3 failures that were fixed in 2d550b9. There was also issues around ulimit values persisting causing test failures in LLVM. Those were addressed in 93dd17a.

boomanaiden154 · 2025-10-23T21:33:49Z

It would be better if someone could fix the test. I don't think anyone else so far has relied on the LD_LIBRARY_PATH pass through in this test. It should work but given it hasn't been validated, there's probably some small detail for why it doesn't.

Support for the external shell is going to be removed eventually and there is a very real risk that new tests do not work with the external shell and there will be very little upstream coverage of that configuration.

pawosm-arm · 2025-10-23T21:37:09Z

It would be better if someone could fix the test. I don't think anyone else so far has relied on the LD_LIBRARY_PATH pass through in this test. It should work but given it hasn't been validated, there's probably some small detail for why it doesn't.

Support for the external shell is going to be removed eventually and there is a very real risk that new tests do not work with the external shell and there will be very little upstream coverage of that configuration.

What worries me more is how low is the coverage of the 'using just-built libc++ for building LLVM' scenario.

DavidTruby

LGTM.

I mainly use the internal shell for lit, but I still regularly see errors I just know to ignore. I don't think it's ready for a product. I do strongly think we should document this somewhere and reconsider in future. I'm also sure issue reports for breaking tests upstream would be much appreciated!

DavidTruby · 2025-10-28T23:55:54Z

What worries me more is how low is the coverage of the 'using just-built libc++ for building LLVM' scenario.

I've also been concerned about this for a long time. There's even supposedly "built-in" bootstrapping builds you can turn on that I've never been able to get to work.

Is there any hope we could ask eg. Linaro to look at a build bot for these?

pawosm-arm · 2025-10-29T16:56:47Z

I'm trying to figure out what issue are we facing here. I've triggered a manual build, and, as expected, it fails on the env.c test case:

env -i LC_ALL=C LD_LIBRARY_PATH="/build/stage/product_build/lib:/build/atfl/lib:/build/atfl/lib/aarch64-unknown-linux-gnu:." CLANG_NO_DEFAULT_CONFIG=1    /build/stage/product_build/bin/clang /llvm/clang/test/Driver
/env.c -### -o /build/stage/product_build/tools/clang/test/Driver/Output/env.c.tmp.o --target=i386-unknown-linux      --sysroot=/llvm/clang/test
/Driver/Inputs/basic_linux_tree      --rtlib=platform --unwindlib=platform -no-pie      2>&1 | /build/stage/product_build/bin/FileCheck --check-prefix=CHECK-LD-32 /llvm/clang/test/Driver/env.c
# executed command: env -i LC_ALL=C 'LD_LIBRARY_PATH=%{readfile:/build/stage/product_build/tools/clang/test/Driver/Output/env.c.tmp.ld_library_path}' CLANG_NO_DEFAULT_CONFIG=1 /dsg_space/projectscratch_dsg_space/pawosm
01/atfl-development/arm-toolchain.git/arm-software/linux/build/stage/product_build/bin/clang /llvm/clang/test/Driver/env.c '-###' -o /build/stage/product_build/tools/clang/test/Driver/Output/env.c.tmp.o --target=i386-unknown-linux --sysroot=/llvm/clang/test/Driver/Inputs/basic_linux_tree --rtlib=platform --unwindlib=platform -no-pie
# note: command had no output on stdout or stderr
# error: command failed with exit status: 127
# executed command: /build/stage/product_build/bin/FileCheck --check-prefix=CHECK-LD-32 /llvm/clang/test/Driver/env.c
# .---command stderr------------
# | /llvm/clang/test/Driver/env.c:20:17: error: CHECK-LD-32: expected string not found in input
# | // CHECK-LD-32: "{{.*}}ld{{(.exe)?}}" "--sysroot=[[SYSROOT:[^"]+]]"
# |                 ^
# | <stdin>:1:1: note: scanning from here
# | /build/stage/product_build/bin/clang: error while loading shared libraries: libc++.so.1: cannot open shared object file: No such file or directory
# | ^
# | <stdin>:1:228: note: possible intended match here
# | /build/stage/product_build/bin/clang: error while loading shared libraries: libc++.so.1: cannot open shared object file: No such file or directory
# |                                                                                                                                                                                                                                    ^
# |
# | Input file: <stdin>
# | Check file: /llvm/clang/test/Driver/env.c
# |
# | -dump-input=help explains the following input dump.
# |
# | Input was:
# | <<<<<<
# |             1: /build/stage/product_build/bin/clang: error while loading shared libraries: libc++.so.1: cannot open shared object file: No such file or directory
# | check:20'0     X~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ error: no match found
# | check:20'1                                                                                                                                                                                                                                        ?                  possible intended match
# | >>>>>>
# `-----------------------------
# error: command failed with exit status: 1

And, having all those files still there, I could confirm that it doesn't happen in the command line (did the positive test case first, that clang actually needs LD_LIBRARY_PATH to be set):

$ /build/stage/product_build/bin/clang
/build/stage/product_build/bin/clang: error while loading shared libraries: libc++.so.1: cannot open shared object file: No such file or directory

$ LD_LIBRARY_PATH="/build/stage/product_build/lib:/build/atfl/lib:/build/atfl/lib/aarch64-unknown-linux-gnu:." /build/stage/product_build/bin/clang
clang: error: no input files

$ env -i LC_ALL=C LD_LIBRARY_PATH="/build/stage/product_build/lib:/build/atfl/lib:/build/atfl/lib/aarch64-unknown-linux-gnu:." CLANG_NO_DEFAULT_CONFIG=1    /build/stage/product_build/bin/clang
clang: error: no input files

Now the questions are:

what is the lit internal shell anyway? do we build something bash-alike for running the check-all tests in LLVM?
is env in lit something other than the system-provided env? Does it filter LD_LIBRARY_PATH to get rid of all of the non-lit-blessed content?

boomanaiden154 · 2025-10-30T00:14:13Z

I think llvm/llvm-project#165140 might fix this.

what is the lit internal shell anyway? do we build something bash-alike for running the check-all tests in LLVM?

Not really bash like. It runs the binaries directly inside Python using subprocess.run instead of writing out a shell script and then executing that.

is env in lit something other than the system-provided env? Does it filter LD_LIBRARY_PATH to get rid of all of the non-lit-blessed content?

Yes. It's a builtin function. It by doesn't do anything special to LD_LIBRARY_PATH though.

pawosm-arm · 2025-10-30T12:32:41Z

I think llvm/llvm-project#165140 might fix this.

I've tried this locally, and it seems it does really help. Thing is, the way this PR has been opened does not make it easy to just cherry-pick it, I had to rework this patch manually in order to try it out.

pawosm-arm · 2025-10-30T21:58:17Z

Closing it as the upstream fix has proven to work.

[ATfL] Make sure internal shell is not used by lit

c98d213

This is to mitigate the effects of this commit: llvm/llvm-project@ee0f86d

pawosm-arm requested review from DavidTruby, davemgreen, garyjcarroll, schander-arm and tblah October 23, 2025 20:50

pawosm-arm requested a review from a team as a code owner October 23, 2025 20:50

DavidTruby approved these changes Oct 28, 2025

View reviewed changes

pawosm-arm closed this Oct 30, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[ATfL] Make sure internal shell is not used by lit #602

[ATfL] Make sure internal shell is not used by lit #602

Uh oh!

pawosm-arm commented Oct 23, 2025

Uh oh!

pawosm-arm commented Oct 23, 2025

Uh oh!

boomanaiden154 commented Oct 23, 2025

Uh oh!

pawosm-arm commented Oct 23, 2025 •

edited

Loading

Uh oh!

DavidTruby left a comment

Uh oh!

DavidTruby commented Oct 28, 2025

Uh oh!

pawosm-arm commented Oct 29, 2025

Uh oh!

boomanaiden154 commented Oct 30, 2025

Uh oh!

pawosm-arm commented Oct 30, 2025

Uh oh!

pawosm-arm commented Oct 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

[ATfL] Make sure internal shell is not used by lit #602

[ATfL] Make sure internal shell is not used by lit #602

Uh oh!

Conversation

pawosm-arm commented Oct 23, 2025

Uh oh!

pawosm-arm commented Oct 23, 2025

Uh oh!

boomanaiden154 commented Oct 23, 2025

Uh oh!

pawosm-arm commented Oct 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DavidTruby left a comment

Choose a reason for hiding this comment

Uh oh!

DavidTruby commented Oct 28, 2025

Uh oh!

pawosm-arm commented Oct 29, 2025

Uh oh!

boomanaiden154 commented Oct 30, 2025

Uh oh!

pawosm-arm commented Oct 30, 2025

Uh oh!

pawosm-arm commented Oct 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

pawosm-arm commented Oct 23, 2025 •

edited

Loading